Skip to content

Use MonitorUpdatingPersister in ChainMonitor - #667

Closed
amackillop wants to merge 18 commits into
lightningdevkit:mainfrom
cequals:amackillop/blt-380/use-mup
Closed

amackillop wants to merge 18 commits into
lightningdevkit:mainfrom
cequals:amackillop/blt-380/use-mup

Conversation

@amackillop

Copy link
Copy Markdown

Replace direct KVStore usage with MonitorUpdatingPersister to enable incremental channel monitor updates instead of rewriting the entire monitor on each update. This improves I/O efficiency by writing only differential updates (typically hundreds of bytes) rather than rewriting the full monitor (potentially tens of megabytes) for each state change. The existing behaviour tends to make channel updates very slow after a few thousand updates.

The persister is configured with a maximum of 100 pending updates before consolidating to a full monitor write. This balances I/O bandwidth savings against storage complexity.

Note: MonitorUpdatingPersister format is not backward-compatible with the standard persister due to the sentinel byte sequence prepended to monitors.

amackillop and others added 18 commits September 23, 2024 14:09
This PR adds a Prober to the Node struct in order to probe the network separate from the payment process
Forgot to make the prober mod public so that the structs could be used by the Prober.

This also handles all events in ldk node directly so that the the Probe events all get handled correctly while that is happening here. Otherwise, there is a race between event handling in the Prober and event handling here which can cause Probes to hang. This is temporary, next step is to figure out how to serialize those Probe events so that we can do this in the Prober.
Make logger configurable with default behavior

Add configuration options for logging
* upstream/main: (85 commits)
  Update Swift files for v0.4.2
  Bump version number to v0.4.2
  Update `CHANGELOG` for v0.4.2
  Add comments and debug assertions for `list_unspent_utxos`
  Account for path changes in vss-server.
  Fix invalid witness program length
  Update CHANGELOG for v0.4.1
  Bump version number to 0.4.1
  fix: incremental and full scans are performed correctly
  Update CHANGELOG.md
  Update Swift files for v0.4.0
  Update CHANGELOG for v0.4
  Bump version number to 0.4.0
  Expose build_with_vss_* methods in bindings.
  Launch VSS, remove cfg flag for it.
  Use KeyObfuscator in VssStore.
  Add build_with_vss* methods for ArcedNodeBuilder.
  Default to LnUrlJWT auth for using VSS.
  Enable using VssStore with VssHeaderProvider.
  Bump kotlin-android `compileSdk` to 34
  ...
* upstream/main: (25 commits)
  spontaneous payment send_with_custom_tlv
  core: fixing the esplora error match
  Drop name of elided lifetime
  Start publishing PaymentForwarded events.
  Drop unnecessary `vec!` allocations in RPC calls
  Relax chain polling interval to 2 seconds
  Poll mempool entries interatively
  Remove pin on `hashlink`
  Bump `rusqlite` to `0.31`
  Add note to handle events as quickly as possible.
  Add `.clippy.toml` to lint MSRV
  Add test for `send_all_to_address` behavior
  Allow honoring reserve in `send_all_to_address`
  Log HTTP 400 at `TRACE` instead of ignoring it
  fix: compile with tokio_unstable
  refactor: return error when creating logger directory
  Pin `indexmap` back to account for `hashbrown` MSRV bump
  docs: add default log filename to documentation
  refactor: replace log directory path to log file path
  Test we can recover on-chain funds from seed-only
  ...
This PR adds scoring parameters to the node configuration. This allows for setting custom scoring parameters which we will use to run experiments. This also adds the methods to import/export/reset the scorer and update its decay parameters. Updating fee parameters requires a restart of the node to update the router
This stops the sending of any payments to the Prober. May also need to restrict channel opens to just c= as well as in theory a random node could also open with push_msat
Adds a simple method to fully sync RGS data ie use timestamp 0
- Remove scorer update method
- Remove attempts field from route finding. .with_max_path_count(attempts); is for multi-path payments
* upstream/main: (53 commits)
  Recalculate listeners' best block heights on each retry
  Reject inbound announced channels if not all requirements are met
  Add logging of RPC syncing logic
  Cache mempool entries and transactions to avoid retransmissions
  Don't error out on dropped mempool entries
  Add pathfinding scores export method
  Don't log value attempted on channel `send` failure
  Update LSPS2 link in payment store
  Add Uniffi bindings for LSPS1 API
  Add LSPS1 API
  Add `create_order` logic
  Only skip bumping channel closes for trusted peers
  Add missing argument to `README` example
  Bump shutdown logging from `TRACE` to `DEBUG`
  Drop redundant reconnection logs
  Bump peer reconnection interval to 1 minute
  Have `set_log_facade_logger` take an *optional* `max_log_level`
  Fix log-facade logging
  f Builder docs
  Rename `level` argument to `max_log_level`
  ...
Replace direct KVStore usage with MonitorUpdatingPersister to enable
incremental channel monitor updates instead of rewriting the entire
monitor on each update. This improves I/O efficiency by writing only
differential updates (typically hundreds of bytes) rather than
rewriting the full monitor (potentially tens of megabytes) for each
state change. The existing behaviour tends to make channel updates
very slow after a few thousand updates.

The persister is configured with a maximum of 100 pending updates
before consolidating to a full monitor write. This balances I/O
bandwidth savings against storage complexity.

Note: MonitorUpdatingPersister format is not backward-compatible
with the standard persister due to the sentinel byte sequence
prepended to monitors.
@ldk-reviews-bot

Copy link
Copy Markdown

👋 Hi! Please choose at least one reviewer by assigning them on the right bar.
If no reviewers are assigned within 10 minutes, I'll automatically assign one.
Once the first reviewer has submitted a review, a second will be assigned if required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants